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This paper examines some of the problems that arise 
when conducting multivariate analyses incomplete data. The 

literature on the effectiveness of several missing data procedures 
(MDP) is summarized. The most widely used MDPs are: (1) listwise 
deletion; (2) pairwise deletion; (3) variable mean; (4) correlational 
methods. No ifflP should be used without considering the 
characteristics of the data at hand, even though the ii terature may 
support one MDP. An application of these MDPs was demonstrated using 
college placement data taken from a published article on predicting 
success in foreign language training. The data included 12 variables 
(10 predictors and 2 criteria) for 279 college students. Descriptive 
statistics were produced by listwise and pairwise deletions, and 
missing predictor values were imputed. Missing data indicator 
variables were created to indicate the presence or absence of values 
on each of the original variables. The matrix of correlations between 
the indicator variables and the original variables appears in Table 
3. The primary analysis involved the use of stepwise regression 
equations based on three. alternative MDPs. The conclusion is made 
that regardless of the MDPselected, it is important to first 
determine the properties of the missing values, and second to decide 
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AN INTERACTIVE APPROACH TO ANALYSING 
INCOMPLETE MULTIVARIATE DATA 

Missing data can be a nuisance in all types of research, particularly in 
studies conducted in applied settings. The problem is compounded in 
multivariate, research because nearly all fonns_of multivariate analysis 
require complete data from all cases; those cases with incomplete data 
are, by default, discarded. A casual review of any literature in the 
applied behavioral sciences, or of toe most widely used statistical 
software .packages, seems to indicate that the most widely used pro- 
cedures for dealing with missing data arer listwise_deletion, pairwise 
deletion, and mean substitution. Alternative procedures , such as 
imputing missing values by regressing variables with missing values onto 
relevant covariates, are rarely applied. 



The purpose of this paper is to examine some of the problems that arise 
when conducting multivariate analyses with incomplete data. First, the 
literature on the effectiveness of several missing data procedures 
(MDPs) is summarized. Second, a rational application of these empirical 
MDPs is demontrated using college placement data. Specifically, it is 
suggested that no procedure should be blindly applied in any instance, 

even though the. literature may strongly support the use of one HDP. 

Third, the results of regression analyses utilizing three different MDPs 
are compared using the same set of college entrance data. The motiva- 
tion for this comparison is to determine the extent of the practical 
differences among MDPs. 

The Effectiveness of MDPs 

Numerous strategies f or dealing with inco^lete multivariate data have 
been proposed over the last 30 years. The most general and accessable 
of these MDPs are listed below. 

1. LISTWISE DELETION. By default most multivariate algorithms 
discard all cases with tecompiete data. The major limita- 
tion to this approach is "Uiat_relevant data are frequently 
discarded. At best, power suffers and Type II error rates 
increase. At worst, bias is introduced. 

2. PAIRWISE DELETION. This procedure utilizes all available 
data-in the cCTaputation of means and variances, _and all 
avaliable pairs of _values for the computation of covar- 
iances. Subsequent analyses (multiple regression) utilize 
toe_resulting supnary statistics. A conceptual limitation 
is that when correlations and other statistics are based on 
different but overlapping subsamples of a larger sample, the 
population to which generalization, is sought is ho longer 
clear. Statistical problems may also_arise when the data 
are missing in a nonrandom fashion. That is, it is possible 
to obtain correlation matrices with mutually inconsistent 
correlations. 
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3. VARIABIE MEM*. Missing values are replaced by the mean as 
computed for all cases that are present . The conceptual 
limitation with the variable mean MDP is that it is hot 
intuitively appealing, or empirically accurate, to insert 
means when the_missing observation in guest ion occurs for a 
subject whose observations on other variables are quite 

distant from toe mean. The empirical limitation is that 

inserting means will attenuate variance and covariahce 
estimates . 

4. CORRELATIONAL METHODS. Missing values are replaced with the 
ah estimate ttatis_cbtained^ regressing the incomplete 
variable onto one_or more coyariates. These. procedures can 
be exeeuted_dif ferent ways^ by varying several factors: A) 
the start tag point _ for computations (e.g., listwise dele- 
tion, pairwise deletion, etc.); B) the number of covariates 
used in the regression equations; C) whether or hot an 
iterative solution is used; D) whether or not the resulting 
variahce-cbvariance matrix is corrected for bias. 

Several simulation studies have shown that obtaining estimates based on 
correlational procedures_ provides results that most closely approximate 
what would have been obtained had the data remained complete. The 
iterative regression algorithms have sometimes demonstrated quite 
dramatic improvements over listwise and pairwise deletion (e.g. , Beaie & 
Little, 1975; Gleason & Staelin, 1975; Chan, Gilman & Dunn, 1976) . When 
the data are reasonably intact (better than 95% complete) , and there are 
fewer than four variables, then the simple tbivariate) regression MDP is 
about as effective as the iterated regression procedure (Raymond & 
Roberts, 1987) . A comprehensive review of the literature on multi- 
variate analysis from incomplete data indicates that the most popular 
and expedient MDPs are, in general, the least effective (Raymond, 1986) . 
In particular, pairwise deletion has been shown to lead tcf some very 
inaccurate results (Haitovsky, 1968) . 

An Approache s Analyzing Incomplete Data 

Although there is ample support for imputing missing values through the 
use of correlational methods, these MDPs should be judiciously applied. 
No MDP should be used without first considering the characteristics of 
the data at hand. For example, using predictors to estimate criteria 
can result in inflated R 2 s in subsequent analyses . Similarly, using two 
predictors to estimate one another may induce unwanted mult icol linear- 
ity. Also, both the amount and the pattern of missing data may have 
implications for how subjects with missing values_are treated, it may 
be necessary to drop some subjects while estimating missing values for 
other subjects^ Similarly, if a particular variable is prone to 
extensive nonresponse, then that variable may need to be excluded from 
analysis. 



The example below, is : intended to demonstrate some of the factors to 
consider when analyzing incomplete multivariate data. It is not 
intended to represent the correct approach, but rather one approach that 
recognizes the fact that aissing data has the potential for altering the 
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conclusions of a study. The remainder of the paper is presented in 
three sections: tile description of the data set, the description and 
treatment of nonresponse, and the execution of the primary analysis. 
The primary analysyis involves a comparison of stepwise regression 
equations computed from data treated by three MDPs. 

Data set 

The data originally appeared in a published article on predicting 
success in foreign language training. The raw data included 12 Vari- 
ables J 10 predictors and 2 criteria) for 279^college students. The 
primary purpose of the analysis was to determine if the prediction of 
performance in a foreign language course could be augmented by addinq 
one or more new variables to the existing prediction system. While not 
all 12 variables are directly pertinent to this analysis, all will be 
useful m the treatment of the missing values. 

The variables include: high school GPA (HS-GPA) , SAT-V and SAT-M scores. 
7 n f -iS Pi acement tes t scores, two subtest scores on the Modern Language 
Aptitude Test (MLAT-4 and HIAT-5) , Foreign language Attitude scores, 
age, sex, prior experience with a foriegn language, current college GPA 
and final grade in a foreign language course (FL Grade) . 

Descri p tion an d explanation of nonrestxanse 

The total sample size included 279 cases. Due to missing data on one or 
more of the variables, the listwise deletion sample size consisted of 
174 subjects, or 62% of the original sample. The pairwise deletion 
sample sizes ranged from 197 to 278. Most of the missing data occurred 
for MLAT-4, MLAT-5, and FL Grade. In all about 9.2% of the original 
data matrix was missing (N=279, p=12, m=308) . 



Of the 279 subjects, two of the students were missing data on six of the 
ten predictors. These two students were elimated from any further 
analysis. Another 45 students had incomplete criterion data (i.e. , ro 
data for FL Grade) , and were excluded. Most of these students were 
audits or drop-outs. Thus the sample size was reduced to N=230. This 
revised data set was 93.9% complete (based on the revised sample of 
N-230, P=12) , as it still consisted of 168 missing values . This 
revised data set was saved and will be restored to completeness by 
estimating the missing values. It is important to note that the entire 
data matrix (N=279) will not be restored; only the subset (N=230) with 
complete data for FL Grade. 

Missing values on FL Grade were not to be estimated from certain 
predictors because the R 2 from the primary analysis would be inflated . 
That is, using MLAT scores to estimate missing Values on FL Grade would 
only serve to artificially increase the relationship between the two 
variables in the final analysis. Further, it did hot seem appropriate 
to estimate FL Grade from College GPA^an irrelevant criterion 
variable) , since the relationship of FL Grade with certain predictors 
may have been lowered, specifically those predictors designed to be 
orthogonal to college GPA while being related to FL Grades, it is worth 
noting that students with missing FL Grades performed slightly lower 
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than other students on HS-GPA, MIAT-4, and SAT-M. The likely conse- 
quence of eliminating these students will be a restriction in range, 
which may attenuate some of the correlations. 

A portiohof Table 1 lists the descriptive statistics produced by 
listwise and pairwise deletion. A comparison of correlation matrices 
revrals some differences in the maginitude of the correlations. For 
example, the conrelation_ between HIAT-4 and FL Grade was .38 using 
listwise deletion and .43 using pairwise deletion. There were also some 
differences in means and standard deviations preluded by the two 
methods. Pairwise deletion resulted in lower means and smaller standard 
deviations produced by on some of the ability predictors suggested that 
lower ability students were more likely to have missing data. Listwise 
deletion seemed to be an undesirable alternative for two reason: loss of 
power and potential bias. Pairwise deletion did not seem to be the 
desirable_ solution for theoretical reasons related to sampling and 
generalization and because of its poor performance in orior simulation 
experiments . 

At this point in the analysis, the imputation of missing predictor 
values seemed a reasonable course of action. Initial estimates were 
first obtained using college-GPA, an otherwise irrelevant variable. 
College GPA was moderately correlated with the ability predictors, and 
thus ^ provided more accurate initial estimates than would have been 
provided by the inserting the variable mean. Next, the initial esti- 
mates were revised using stepwise regression for two iterations . in 
all, 168 blanks on_eight variables were_replaced by estimates, with the 
eight multiple Rs ranging from .27 to .79, with a mean multiple R of 
about .60. The descriptive statistics provided by the restored data 
matrix_also appear in Table 1. Again, there are some notable differ- 
Br cbs among correlations produced by the three MDPs. A checkmark [/) 
indicates correlations that differ by at least ± .05. 

aide to the further investigation of nonresponse, missing data 
indicator variables were created to indicate the presence or absence of 
values on each of the original variables. For example, if a subject 
failed to respond to the item concerning Age, they were assigned a value 
of 1 on a new variable call^i_#Age. The descriptive statistics _for the 
missing data indicator variables revealed a few important character- 
istics of the data. The sum of observations (1 or 0 over N subjects) 
corresponded to the number of missing values, while the mean_reflected 
the proportion of students with missing data. ThB correlations 
indicated whether or hot there was a general tenden^ for nonresponse to 
some of the variables. As indicated in Table 2, some of the correla- 
tions among the indicator variables were .98 and 1.00. 

[AN ASIDE: A principal components analysis_of the matrix in 
Table 2 resulted in a_pattern of loadings that corresponded 
to the manner in_which the da^a for_ this study were collect- 
ed. The_data for this study were collected from two sources: 
College records and measures collected during two sessions 
that occured during class time. The loadings on the first 
exponent corresponded to missing data on variables obtained 
from college records. Variables loading on the second 
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component were those collected during the two class sessions 
(MEAT scores; demographies and attitudes); Missing data on 
these variables occured for students w ho elected not to 
participate in the study. The third component was bipolar, 
and corresponded to students who provided demographic and 
attitudinal information but did not take the MIAT. The 
principal components analysis was hot necessary in the 
present context because the data set was small enough, and 
the missing values few enough, to obtain the same information 
by a visual inspection of the missing data matrix. However, 
such an analysis would be quite helpful for large data sets.] 

Table 3 presents the matrix of correlations between the indicator 
variables and the original variables. Several moderate correlations 

scoring lower on many of the 
academic performance measures were more likely to have missing Values. 
A three variable regression equation consisting of Age , MIAT-4 , and 
College GPA predicted "missingness" on college records (SAT, HS-GPA and 
English Placement) with a multiple correlation of .47. It is obvious 
that the data are missing in a no^random fashion; thus general izability 
of th& regression equations obtained from the orimary analysis will be 
limited. 

Execution of Primary Analysis 

The primary data analysis involved the use of stepwise regression to 
determine the effectiveness of ability, demographic, and attitudinal 
variables in predicting performance in foreign language courses. Here 
we will compare three regression equations based on three alternative 
MDPs: jpairwise deletion (N=174) , listwise deletion (278 <N< 197) , and 
restoration by imputing missing values (N=230) . The data based on 
listwise deletion resulted in a five variable equation that produced an 
R 2 of ' 35 ' Pairwise deletion produced a four variable equation with an 
R of .29. The restored data gave a different arrangement of the same 
variables which produced an R 2 of .30. The standardized regression 
equations appear below. 

LISTWISE DELETION: R 2 =.354 

FL Grade = HS-GPA (.351) + FL Att(.211) + Age (.252) + 
MIAT-4 (.231) + Prior Exp (.164) 

PAIRWISE DELETION: R-=. 291 

FL Grade = MIAT-4 (.275) + HS-GPA (.2 62) + FL Att(.l92) + 
Prior Exp (.123) 

RESTORATION: R 2 =.,199 

FL Grade = HS-GPA(.309) + FL Att(.196) + MLAT-4(.252) + 
Age (.163) + Prior Exp(.147) 
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Conduct ing Coasaenfes 

Although it may be argued that no analysis should be conducted when the 
level of miss ing_ date exceeds a certain percentage (pay 5%) , the fact 
remains that such studies are often reported in the literature. 
Conduct ing_anal^es on incoitplete data can be justified by the argument 
that^the less than precise information provided by studies utilizing 
fragmentary samples is preferable to no information at all. 

The three MDPs provided somewhat diixerent findings. The interpretation 
of the reuits^hould be guided by the characteristics of the missing 
data._ The correlations in Table 3 can be useful for this. Specifical- 
ly, the results may have limited appl icabil ity to older students . The 
prediction model developed from listwise deletion completely ignored the 
students with incomplete data (many of whom were older or lower in 
academic aptitude)^ Iherefore f any report based on the listwise 
deletion MDP shouldment ion this limitation. It is difficult to specify 
how the results based on pairwise deletion should be interpreted. The 
level of incompleteness (9.2%), and the systematic nature of the missing 
data would urge an extremely cautious interpretation of the regression 
equations. 

The gCTeraiizabili^r of the regression equations based oh the restored 
data matrix depends on the accuracy with which the missing values have 
been ±ir5>uted<^ ^ ^ a m^ianum, there will be a restriction of range due to 
the i^^ed_values being regressed toward the mean. Corrections for 
this type of systematic error are discussed in Beale and Little (1975) , 
and tittle (±978±* _ Even more serious errors may result if the 
relationships between the incomplete variables and ttidse variables used 
to estimate themissing values are not linear oyer the full _ range of 
values. __ln such ^stances, missing values may be drastically under- 
estimated or overestimated. Regression diagnostics can be useful for 
assessing the ijnpact of the imputed values on the prediction models 
obtained from restored data matrices. In the current restored data, 
there was a tendency for those cases with missing values to be 
qverpredicted by the regression model (a preponderance of negative 
residuals) . in addition, there were three cases flagged with high 
leverage indexes. Each of these three cases were individuals with 
incomplete data. 

Which MDP should be trusted for the current analysis? While the results 
of several Monte Carlo studies bffsr fairly strong support for the use 
of correlational methods to impute missing values , one may be inclined 
to feel a_ little uncomfortable with fabricating data, even if it Is done 
empirically. Adopting such a position requires that some other meaning- 
ful strategy be used to reach a decision. (We can readily acknowledge 
that choosing the HDP that gives the highest value of R 2 is not 
acceptable I ) Examining the regression statistics for the three MDPs for 
consistency may provide some indication of which MDP should be chosen, 
or which should be avoided -. 

The differences in the regression weights between iistwise_ deletion and 
restored data were minor, even Qiough the JR 2 values were notably 
different -. Oh the other hand, pairwise deletion and estimation by 



8 

o 

ERIC 



«4 



7 

regression provide similar values of R^, even though recession 
equation provided by pairwise deletion was substantially different from 
the regression equations provided by the other two MDPs. Given that the 
goal in conducting regression analyses bh college entrance data is to 
develop empirical models which can be applied to_future samples, it 
seems that more credence should be given to the consistency of beta 
weights. This line of reasoning rules but pairwise deletion. Choosing 
between listwise deletion and the restored data shouid_be a function of 
the palliation to which general izability is sought, and the degree of 
bias that may have been introduced by each of the two MDPs. 

Regardless of the MDP selected, it is important _to first determine the 
properties of the missing values, and second, to overtly decide which 
MDP seems most prudent. Clearly, hb MDP is a perfectly reliab?,e 
substitute for complete data. However, when missing data do occur, it 
xs usually the case that some MDP is utilized, even if by default. 
Listwise deletion and pairwise deletion, while sometimes appropriate, 
should not be relied on simply as a matter of expedience. 
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Table 1 

DESCRIPTIVE STATISTICS OBTAINED FROM THREE KISSING DATA STRATEGIES 



Age Sex 



Prior FL MUST HLAT US Eng Ft college 

Exper Atti 4 5 CPA SAT-V SAT-M Place Grade GPA 



N MEAN S D 



Age 



Sex 



Prior Exper 



FL Attitude 



NLAT ■ 4 



HLAT • 5 



US GPA 



SAT • V 



SAT : I 



Eng Place 



FL Grade 



College GPA 
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•06/ 
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17 
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■07 
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■14 
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23 
23 
23 


05 
08 
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03 
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■04 « 
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•06 
•09 


00 
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: 22 
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•06 
•08 
•10 


12 
11 
10 


•08 
•06 
•08 


■02 J 
01 


15 
18 

r 


is 

13 

12 


16 . 
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11 


02 
03 
02 


: 11 



100 
100 
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Listwi se Deletion 
Pairwise Deletion 



Estimation by / 
Linear Regression 
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07 



100 

loo 



22 
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06 
05 



37 
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44 
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0:65 


» 268 


1:72 


0.87 




1.73 


0.83 


174 


0:46 


0:50 


278 


0.45 


0.50 


230 


0.45 


0.50 


174 


3.19 


1.60 


268 


3.21 


1.60 


230 


3.23 


1.58 


174 


83.16 


12.64 


279 


82.49 


14.03 


230 


83.21 


13.09 


174 


25:31 


6:11 


230 


24.26 


6.23 


230 


24.67 


P A«\ 

5.92 


174 


19.69 


4.85 


230 


19.24 


4.?2 


230 


19.37 


4.63 


174 


3.36 


0.46 


245 


3.29 


0.48 


230 


3:31 


0:45 


174 


507.36 


89.20 


245 


501.51 


91.16 


230 


507.53 


86.20 


174 


575.93 


85.32 


245 


564.25 


88.71 


230 


C7* - mi 

570.71 


tit ti 

83.61 


174 


_ . ... 
54.92 


._ _.. 
15.6.0 


242 


53.95 


15:40 


230 


54.39 


14.89 


174 


3.40 


0:85 


232 


3.32 


0:88 


230 


3.34 


0.87 


174 


2.84 


0.61 


278 


2.75 


0.62 


230 


2.77 


0.62 



Table 2 



CORRELATIONS BETWEEN HISSING DATA INDICATOR VARIABLES 



#?rior #FL MAT MAT #HS .... #Ehg. JR.. College Percent Number 
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100 
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14 
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10 


10 


•13 


•13 


100 










10.0 


23 
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1Q 


10 


•13 


•13 


100 


100 
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23 


SSAT - H 
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10 


•13 


•13 
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10.0 


23 
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09 


09 


•13 


•13 


98 


98 


98 
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24 
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0.0 


0 


{(College GPA 


•01 '. 


•01 ! 


•03 


•03 


20 


20 


20 


19 ! 
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0.4 
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Table 3 

CORRELATIONS BETWEEN ORIGINAL VARIABLES 
HISSING DATA INDICATOR VARIABLES 





Age 


Sex 


Prior 
Exper 


FL 

Atti 


HLAT 

4 


HLAT 
5 


US 
GPA 


SAT-V 


SAT-H 


Jng 
Place 


FL 

Grade 
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GPA 




















#Age 


•05 


07 


04 


12 


02 


05 


•03 


•05 


00 


03 


•05 


•08 


#Sex 


























ifPribr Exper 


; 05 


07 


04 


12 


02 


05 


•03 


•05 


00 


03 


•05 


•08 


fEL Attitude 


























IHLAT • 4 


•03 


•00 


09 


•05 


•04 


•03 


•11 


04 


•07 


•00 


'oo 


•11 


IHLAT • 5 


•03 


•00 


09 


•05 


•04 


•03 


•11 


04 


•07 


•00 


00 


•11 


IHS GPA 


41 


•04 


•02 


•01 


•26 


•18 


■16 


•00 


•09 


•09 


•23 


•15 
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41 


•04 


•02 


•01 


•26 


•18 


•16 


•00 


•09 


•09 


•23 


•15 
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41 


•04 


•02 


•01 


•26 


•18 


•16 


•00 


•09 


•09 


•23 


•15 


lEng Place 


39 


•05 


•04 


00 


•26 


•19 


•17 


01 


•07 


•08 


•22 


•« 


IFL Grade 


























ICollege GPA 


02 


•06 


•09 


06 


02 


•05 


•02 


01 
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NOTES: 1. indicator variables are those preceded by a t symbol . 0 = DATS PRESENT, 1 = OATS HISSING. 
2. Correlations in Tables 2 and 3 are based on N = 230 jndiyiduats with i final grades in FLcourse. 
3; The missing entries in Tables 2 and_3.are.due to complete data for SEX; FL ATTITUDE; and. Ft GRADE for the 230 cases. 
5, Prior to computing correlations in Table 3, missing values were replaced by estimates obtained from multiple 
regression. Without estimates it possible to obtain the correlations that occupy the 

diagonal (e.g.; a correlation between age and Sage). 



